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IN THE CLAIMS 

1. (Previously Presented) A computer readable storage medium having instructions, which when 
implemented by a computer generate client side markup for a client in a server/client system, the 
instructions comprising: 

a set of controls defined in an authoring page for a website for defining desired visual 
renderings and at least one of recognition and audible prompting on a client in a 
server/client system, each control having a first set of attributes directly related to 
visual rendering and a second set of attributes directly related to at least one of 
recognition and audibly prompting, the controls being related to client side markup 
executable by a client browser; and 

a module operable on a computer, the module being configured to receive the authoring 
page, and wherein the module is further configured to generate, using modality 
dependent attributes provided directly from the controls on the authoring page, 
client side markup executable by the client browser on the client in the server/client 
system in accordance with the controls and the attributes of the controls to perform 
both visual rendering and at least one of recognition and audibly prompting. 

s 

2. (Previously Presented) The computer readable storage medium of claim 1 wherein one of the 
second set of attributes for one of the controls relates to a grammar to use for recognition. 

3. (Previously Presented) The computer readable storage medium of claim 2 wherein said one of 
the second set of attributes provides a reference to a location of the grammar. 

4. (Previously Presented) The computer readable storage medium of claim 2 wherein the grammar 
is for one of speech recognition, handwriting recognition, gesture recognition and visual 
recognition. 



5. (Previously Presented) The computer readable storage medium of claim 4 wherein the controls 
relate to one of HTML, XHTML, cHTML, XML and WML. 

6. (Previously Presented) The computer readable storage medium of claim 1 wherein the controls 
relate to one of HTML, XHTML, cHTML, XML and WML. 

7. (Previously Presented) The computer readable storage medium of claim 1 wherein one of the 
second set of attributes for one of the controls provides instructions related to generating audible 
output. 

8. (Previously Presented) The computer readable storage medium of claim 7 wherein the 
instructions comprise text and the attribute relates to converting the text to audible output. 

9. (Previously Presented) The computer readable storage medium of claim 1 wherein one of the 
second set of attributes for one of the controls relates to a location of data for audible output. 

10. (Previously Presented) The computer readable storage medium of claim 9 wherein the data 
comprises a prerecorded audio data file and the attribute relates to playing the prerecorded audio 
data file. 

1 1 . (Previously Presented) The computer readable storage medium of claim 9 wherein the data 
comprises text and the attribute relates to converting the text to audible output. 

12. (Currently Amended) A computer readable storage medium having instructions defined on an 
authoring page for a website, which when implemented by a computer generate client side markup 
for a client in a server/client system, the instructions comprising: 

a first set of medt^ity-depewdef^visual controls having attributes difeefly-related to 
a first modality of interaction with a user of the client that being defining 



desired visual renderings on the client device, the first set of controls being 
related to client side markup executable by a client browser and 
a second set of medaKty dependent-controls having attributes dH=ee#y-related to a 
second modality of interaction with a user of the client that being ri efifliftg 
de s i r e d ope mt letKifrttifretieHt de v ice ? c ompr isi ng-at least one of recognition 
and audibly prompting, wherein attributes related to audible prompting 
include at least one of inline text for text-to-speech conversion, location of 
data for audible rendering and playing of a prerecorded audio file, the second 
set of controls using at least one of the first controls, the second set of 
controls being related to client side markup executable by a client browser; 
and 

a module operable on a computer, the module being configured to receive the 
authoring page, which includes a plurality of the second set of controls, 
wherein the module is further configured to process the plurality of the 
second set of controls from the authoring page to generate client side 
markup directly from the modality dependent controls that is executable by 
the client browser on the client in the server/client system in accordance 
with the second set of controls and the attributes of the second set of controls 
for at least one of recognition and audibly prompting, and wherein the 
module is configured to use at least one of the first set of controls from the 
authoring page in order to generate markup directly therefrom when 
processing each of the second set of controls. 

13. (Previously Presented) The computer readable storage medium of claim 12 wherein one of the 
attributes for the second set of controls relates to a grammar to use for recognition. 

14. (Previously Presented) The computer readable storage medium of claim 13 wherein said one 
of the attributes provides a reference to a location of the grammar. 
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15. (Previously Presented) The computer readable storage medium of claim 13 wherein the 
grammar is for one of speech recognition, handwriting recognition, gesture recognition and visual 
recognition. 

16. (Previously Presented) The computer readable storage medium of claim 15 wherein the first 
set of controls and the second set of controls relate to one of HTML, XHTML, cHTML, XML and 
WML. 

17. (Previously Presented) The computer readable storage medium of claim 12 wherein the 
controls relate to one of HTML, XHTML, cHTML, XML and WML. 

18. (Previously Presented) The computer readable storage medium of claim 12 wherein one of the 
second set of attributes for one of the controls provides instructions related to generating audible 
output. 

19. (Previously Presented) The computer readable storage medium of claim 18 wherein the 
instructions comprise text and the attribute relates to converting the text to audible output. 

20. (Previously Presented) The computer readable storage medium of claim 12 wherein one of the 
second set of attributes for one of the controls relates to a location of data for audible output. 

21 . (Previously Presented) The computer readable storage medium of claim 20 wherein the data 
comprises a prerecorded audio data file and the attribute relates to playing the prerecorded audio 
data file. 

22. (Previously Presented) The computer readable storage medium of claim 20 wherein the data 
comprises text and the attribute relates to converting the text to audible output. 



23. (Currently Amended) A computer readable storage medium having instructions, which when 
implemented by a computer generate client side markup for a client in a server/client system, the 
instructions comprising: 

a first set of visual controls defined on an authoring page for a website having 

attributes related to a first modality of interaction with a user of the client 
that being visual renderings on the client device, the first set of controls 
being related to client side markup executable by a client browser; and 
a second set of controls defined on the authoring page for defining desired operation 
on the client device having attributes related to a second modality of 
interaction with a user of the client that being at least one of recognition and 
audible prompting, wherein attributes related to recognition include at least 
one of location of grammar for use in recognition and confidence level 
thresholds for recognition and wherein attributes related to audible 
prompting include at least one of inline text for text-to-speech conversion, 
location of data for audible rendering and playing of a prerecorded audio 
file, the second set of controls being selectively associated with the first set 
of controls and wherein values of the second set of controls are synchronized 
with the first set of visual controls , and the second set of controls being 
related to client side markup executable a client browser; and 
a module operable on a computer, the module being configured to receive the 
authoring page, wherein the module is further configured to process the 
controls of the first set in the authoring page to generate client side markup 
for the first modality of interaction by incorporating the attributes in the 
controls that is executable by the client browser on the client in the 
server/client system in accordance with the controls of the first set and the 
attributes of the controls of the first set to perform both visual rendering, and 
wherein the module is further configured to process the controls of the 
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second set to generate client side markup for the second modality of 
interaction by incorporating the attributes in the controls that is executable 
by the client browser on the client in the server/client system in accordance 
with the controls of the second set and the attributes of the controls of the 
second set in the authoring page to perform at least one of recognition and 
audibly prompting. 

24. (Previously Presented) The computer readable storage medium of claim 23 wherein one of the 
attributes for the second set of controls relates to a grammar to use for recognition. 

25. (Previously Presented) The computer readable storage medium of claim 24 wherein said one 
of the attributes provides a reference to a location of the grammar. 

26. (Previously Presented) The computer readable storage medium of claim 24 wherein the 
grammar is for one of speech recognition, handwriting recognition, gesture recognition and visual 
recognition. 

27. (Previously Presented) The computer readable storage medium of claim 26 wherein the first 
set of controls and the second set of controls relate to one of HTML, XHTML, cHTML, XML and 
WML. 

28. (Previously Presented) The computer readable storage medium of claim 23 wherein the 
controls relate to one of HTML, XHTML, cHTML, XML and WML. 

29. (Previously Presented) The computer readable storage medium of claim 23 wherein one of the 
second set of attributes for one of the controls provides instructions related to generating audible 
output. 
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30. (Previously Presented) The computer readable storage medium of claim 29 wherein the 
instructions comprise text and the attribute relates to converting the text to audible output. 

3 1 . (Previously Presented) The computer readable storage medium of claim 23 wherein one of the 
second set of attributes for one of the controls relates to a location of data for audible output. 

32. (Previously Presented) The computer readable storage medium of claim 31 wherein the data 
comprises a prerecorded audio data file and the attribute relates to playing the prerecorded audio 
data file. 

33. (Previously Presented) The computer readable storage medium of claim 31 wherein the data 
comprises text and the attribute relates to converting the text to audible output. 

34. (Previously Presented) The computer readable storage medium of claim 23 wherein one of the 
attributes of the second set of controls relates to an identifier of the associated control of the first set 
of controls to form the association. 

35. (Previously Presented) The computer readable storage medium of claim 23 wherein one of the 
attributes relates to whether the associated control of the second set is available for activation. 

36. (Previously Presented) The computer readable storage medium of claim 35 wherein activation 
relates to generating markup. 

37. (Previously Presented) The computer readable storage medium of claim 35 wherein activation 
relates to execution on the client. 

38. (Previously Presented) The computer readable storage medium of claim 23 wherein the second 
set of controls activates another control of the second set. 
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39. (Previously Presented) The computer readable storage medium of claim 23 wherein the second 
set of controls comprise: 

a question control for generating markup related to audible prompting of a question; and 
an answer control for generating markup related to a grammar for recognition. 

40. (Previously Presented) The computer readable storage medium of claim 39 wherein the 
question control activates the answer control. 

41 . (Previously Presented) The computer readable storage medium of claim 40 wherein the answer 
control includes a mechanism to associate a received result with one of the first set of controls. 

42. (Previously Presented) The computer readable storage medium of claim 41 wherein 
mechanism includes binding the recognition value. 

43. (Previously Presented) The computer readable storage medium of claim 42 wherein the 
mechanism includes issuing an event related to operation of binding. 

44. (Previously Presented) The computer readable storage medium of claim 40 wherein the second 
set of controls comprise: 

a command control for generating markup related to a grammar for one of navigation in 
the markup, help with a task, and repeating an audible prompt. 

45. (Previously Presented) The computer readable storage medium of claim 40 wherein the second 
set of controls comprise: 

a confirmation control for generating markup related to confirming that a recognized 
result is correct. 
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46. (Previously Presented) The computer readable storage medium of claim 45 wherein the 
confirmation control includes an attribute related to the recognized result to be confirmed. 

47. (Previously Presented) The computer readable storage medium of claim 46 wherein the answer 
control includes an attribute related to a confidence level. 

48. (Previously Presented) The computer readable storage medium of claim 46 wherein the 
confirmation control is activated as a function of a confidence level of a received result. 

49. (Previously Presented) The computer readable storage medium of claim 48 wherein the 
confirmation control activates an accept control to accept the recognized result. 

50. (Previously Presented) The computer readable storage medium of claim 48 wherein the 
confirmation control activates a deny control to deny the recognized result. 

51 . (Previously Presented) The computer readable storage medium of claim 48 wherein the 
confirmation control activates a correct control to correct the recognized result. 

52. (Previously Presented) A computer implemented method for defining a website application on 
a server in a server/client architecture, the website application providing markup to a client for 
performing recognition and/or audible prompting on the client, the method comprising: 

defining the website application by creating an authoring page with a first set of 

visual controls having attributes related to a first modality of interaction with 
a user of the client that being visual rendering on the client device, and with 
a second set of controls having attributes related to a second modality of 
interaction with the user of the client that being at least one of recognition 
and audibly prompting, wherein attributes related to audible prompting 
include at least one of inline text for text-to-speech conversion, location of 
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data for audible rendering and playing of a prerecorded audio file, and 
wherein the first set of controls and the second set of controls being related 
to client side markup executable by a client browser, wherein defining 
includes selectively associating controls of the second set of controls with at 
least one control of the first set of visual controls; and 
processing the controls on the authoring page to generate client side markup 

including processing of the first set of visual controls to generate client side 
markup for the first modality of interaction executable by the client browser 
on the client in the server/client system in accordance with the controls of 
the first set and the attributes of the controls of the first set to perform visual 
rendering, and processing the controls of the second set to generate client 
side markup for the second modality of interaction by incorporating the 
attributes in the controls of the second set to perform at least one of 
recognition and audibly prompting. 

53. (Withdrawn) The computer implemented method of claim 52 wherein each of the controls of 
the second set include an identifier attribute for identifying a control of the first set of visual 
controls, and wherein associating includes providing an identifier of at least one control of the first 
set of controls in the corresponding identifier attribute of each of the second set of controls. 

54. (Withdrawn) The computer implemented method of claim 52 wherein the second set of 
controls includes a question control related to audible prompting of a question, and an answer 
control related to a grammar for recognition; and wherein defining the website application with a 
second set of controls related to at least one of recognition and audibly prompting includes 
associating the answer control with the question control. 

55. (Withdrawn) The computer implemented method of claim 54 wherein the second set of 
controls includes a confirmation control related to confirming that a recognized result is correct; 
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and wherein defining the website application with a second set of controls related to at least one of 
recognition and audibly prompting includes associating the confirmation control with a recognized 
result to be received. 

56. (Withdrawn) The computer implemented method of claim 55 wherein the second set of 
controls includes a command control for generating markup related to a grammar for one of 
navigation on the computer, help with a task, and repeating an audible prompt; and wherein 
defining the website application with a second set of controls related to at least one of recognition 
and audibly prompting includes associating the command control with a question control. 



